{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img alt=\"QTPyLib\" align=\"right\" src=\"http://qtpylib.io/docs/latest/_images/qtpy-logo.png\" style=\"margin:10px 10px 0;width:250px\">\n",
    "\n",
    "\n",
    "# Data Workflow\n",
    "\n",
    "\n",
    "QTPyLib: Quantitative Trading Python Library<br>\n",
    "https://github.com/ranaroussi/qtpylib\n",
    "\n",
    "Copyright 2016 Ran Aroussi\n",
    "\n",
    "\n",
    "---\n",
    "<code>Licensed under the GNU Lesser General Public License, v3.0 (the \"License\"); You may not use this file except in compliance with the License. You may obtain a copy of the License at https://www.gnu.org/licenses/lgpl-3.0.en.html\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.\n",
    "</code>\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overview\n",
    "\n",
    "This Jupyter Notebook aims to provide developers with a quick overview on the new `Workflow` module, introduced in version `1.5.47a` or QTPyLib.\n",
    "\n",
    "The `Workflow` module includes some handly methods for working with external data sources when backtesting, and is planned to include methods for analyzing your backtest results in the near future.\n",
    "\n",
    "\\* Please refer to the [Workflow API Documentation](http://qtpylib.io/docs/latest/api.html#workflow-api) for complete list of available methods and parametrs.\n",
    "\n",
    "---\n",
    "\n",
    "Let's start by checking out QTPyLib's version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.5.47a'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import qtpylib\n",
    "qtpylib.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need to import the `Workflow` module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from qtpylib import workflow as wf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Working with External Data\n",
    "\n",
    "Sometimes, you want to backtest your strategies using market data you already have from sources other than the Blotter. Before you can use market data from external data sources, you'll need to convert it into a QTPyLib-compatible data format.\n",
    "\n",
    "Once the data is converted, it can be read by your strategies as CSV files. You can also save the converted data in your Blotter's MySQL database so it can be used just like any other data captured by your Blotter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### 1.1. Downloading Data\n",
    "\n",
    "The `Workflow` module includes 4 methods for downloading market data from either Yahoo! Finance, Google or Interactive Brokers. The methods are:\n",
    "\n",
    "- `get_data_yahoo()` - downloads daily bars from Yahoo! finance\n",
    "- `get_data_yahoo_intraday()` - downloads 1-min intraday bars from Yahoo! finance (2 weeks max)\n",
    "- `get_data_google_intraday()` - downloads 1-min intraday bars from Google (3 weeks max)\n",
    "- `get_data_ib()` - downloads 1s or higher bars from Interactive Brokers (requires a running Blotter, connected to TWS/IB Gateway)\n",
    "\n",
    "---\n",
    "\n",
    "In this example, let's first download intraday market data for the S&P E-mini Futures from Yahoo! finance, so we'll have some data to work with..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>close</th>\n",
       "      <th>high</th>\n",
       "      <th>low</th>\n",
       "      <th>open</th>\n",
       "      <th>volume</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>timestamp</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2016-12-16 09:09:00-05:00</th>\n",
       "      <td>2265.0</td>\n",
       "      <td>2266.75</td>\n",
       "      <td>2264.75</td>\n",
       "      <td>2266.50</td>\n",
       "      <td>700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-12-16 09:14:00-05:00</th>\n",
       "      <td>2265.5</td>\n",
       "      <td>2265.75</td>\n",
       "      <td>2264.75</td>\n",
       "      <td>2264.75</td>\n",
       "      <td>900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-12-16 09:19:00-05:00</th>\n",
       "      <td>2265.0</td>\n",
       "      <td>2265.75</td>\n",
       "      <td>2264.75</td>\n",
       "      <td>2265.25</td>\n",
       "      <td>1100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-12-16 09:24:00-05:00</th>\n",
       "      <td>2265.0</td>\n",
       "      <td>2265.25</td>\n",
       "      <td>2264.75</td>\n",
       "      <td>2265.25</td>\n",
       "      <td>900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-12-16 09:29:00-05:00</th>\n",
       "      <td>2268.0</td>\n",
       "      <td>2268.00</td>\n",
       "      <td>2265.25</td>\n",
       "      <td>2265.25</td>\n",
       "      <td>4600</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            close     high      low     open  volume\n",
       "timestamp                                                           \n",
       "2016-12-16 09:09:00-05:00  2265.0  2266.75  2264.75  2266.50     700\n",
       "2016-12-16 09:14:00-05:00  2265.5  2265.75  2264.75  2264.75     900\n",
       "2016-12-16 09:19:00-05:00  2265.0  2265.75  2264.75  2265.25    1100\n",
       "2016-12-16 09:24:00-05:00  2265.0  2265.25  2264.75  2265.25     900\n",
       "2016-12-16 09:29:00-05:00  2268.0  2268.00  2265.25  2265.25    4600"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get data from Yahoo! finance\n",
    "df = wf.get_data_yahoo_intraday(\"ESZ16.CME\")\n",
    "df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, if you already have your data as CSV files, simply load them into a `pd.DataFrame` that we will convert into QTPyLib-compatible data format in the next step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# read data from existing csv file\n",
    "# df = pd.read_csv(\"~/Desktop/existing_data.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### 1.2. Convert data into a QTPyLib-compatible data format\n",
    "\n",
    "Once you have your existing data loaded as a `pd.DataFrame`, it's time to convert it into a QTPyLib-compatible data format by using the `prepare_data()` method and passing our data and the **IB-Compatible contract tuple or string** as the first argument.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>close</th>\n",
       "      <th>high</th>\n",
       "      <th>low</th>\n",
       "      <th>open</th>\n",
       "      <th>volume</th>\n",
       "      <th>symbol</th>\n",
       "      <th>symbol_group</th>\n",
       "      <th>asset_class</th>\n",
       "      <th>expiry</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>datetime</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2016-12-16 14:09:00+00:00</th>\n",
       "      <td>2265.0</td>\n",
       "      <td>2266.75</td>\n",
       "      <td>2264.75</td>\n",
       "      <td>2266.50</td>\n",
       "      <td>700</td>\n",
       "      <td>ESZ2016_FUT</td>\n",
       "      <td>ES_F</td>\n",
       "      <td>FUT</td>\n",
       "      <td>2016-12-16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-12-16 14:14:00+00:00</th>\n",
       "      <td>2265.5</td>\n",
       "      <td>2265.75</td>\n",
       "      <td>2264.75</td>\n",
       "      <td>2264.75</td>\n",
       "      <td>900</td>\n",
       "      <td>ESZ2016_FUT</td>\n",
       "      <td>ES_F</td>\n",
       "      <td>FUT</td>\n",
       "      <td>2016-12-16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-12-16 14:19:00+00:00</th>\n",
       "      <td>2265.0</td>\n",
       "      <td>2265.75</td>\n",
       "      <td>2264.75</td>\n",
       "      <td>2265.25</td>\n",
       "      <td>1100</td>\n",
       "      <td>ESZ2016_FUT</td>\n",
       "      <td>ES_F</td>\n",
       "      <td>FUT</td>\n",
       "      <td>2016-12-16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-12-16 14:24:00+00:00</th>\n",
       "      <td>2265.0</td>\n",
       "      <td>2265.25</td>\n",
       "      <td>2264.75</td>\n",
       "      <td>2265.25</td>\n",
       "      <td>900</td>\n",
       "      <td>ESZ2016_FUT</td>\n",
       "      <td>ES_F</td>\n",
       "      <td>FUT</td>\n",
       "      <td>2016-12-16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016-12-16 14:29:00+00:00</th>\n",
       "      <td>2268.0</td>\n",
       "      <td>2268.00</td>\n",
       "      <td>2265.25</td>\n",
       "      <td>2265.25</td>\n",
       "      <td>4600</td>\n",
       "      <td>ESZ2016_FUT</td>\n",
       "      <td>ES_F</td>\n",
       "      <td>FUT</td>\n",
       "      <td>2016-12-16</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            close     high      low     open  volume       symbol symbol_group asset_class      expiry\n",
       "datetime                                                                                                              \n",
       "2016-12-16 14:09:00+00:00  2265.0  2266.75  2264.75  2266.50     700  ESZ2016_FUT         ES_F         FUT  2016-12-16\n",
       "2016-12-16 14:14:00+00:00  2265.5  2265.75  2264.75  2264.75     900  ESZ2016_FUT         ES_F         FUT  2016-12-16\n",
       "2016-12-16 14:19:00+00:00  2265.0  2265.75  2264.75  2265.25    1100  ESZ2016_FUT         ES_F         FUT  2016-12-16\n",
       "2016-12-16 14:24:00+00:00  2265.0  2265.25  2264.75  2265.25     900  ESZ2016_FUT         ES_F         FUT  2016-12-16\n",
       "2016-12-16 14:29:00+00:00  2268.0  2268.00  2265.25  2265.25    4600  ESZ2016_FUT         ES_F         FUT  2016-12-16"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# prepare data for usage by QTPyLib\n",
    "ibtuple = (\"ES\", \"FUT\", \"GLOBEX\", \"USD\", 201612, 0.0, \"\")\n",
    "qtpylib_df = wf.prepare_data(ibtuple, data=df, output_path=\"~/Desktop/\")\n",
    "qtpylib_df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The resulting CSV file will be saved in `~/Desktop/ESZ2016_FUT.csv`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### 1.3. Storing Converted Data in MySQL\n",
    "\n",
    "While this step is optional, you may want to load your converted data into QTPyLib's MySQL for future use by your strategies. **This step isn't required as you can backtest directly off of your converted CSV files.**\n",
    "\n",
    "With your `Blotter` running and connected to TWS/IB Gateway, run this command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# store data\n",
    "wf.store_data(qtpylib_df, kind=\"BAR\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### 1.4. Using CSV files when Backtesting\n",
    "\n",
    "Once you have your CSV files in a QTPyLib-compatible format, you can backtest using this data using the `--data` flag when running your backtests.\n",
    "\n",
    "Example:\n",
    "\n",
    "```\n",
    "$ python strategy.py --backtest --start 2016-12-01 --end 2016-12-31 --data ~/Desktop/ -output ~/portfolio.pkl\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "Please refer to [Back-Testing Using QTPyLib](http://qtpylib.io/docs/latest/algo.html#back-testing-using-qtpylib) for more information about back-testing using QTPyLib."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
